feat(processes): start processes on events in the same transaction, with tasks and processes in core - #227

Summary

Two changes to processes and tasks.

1. A change that should start a process now starts it inside the same transaction, so the run commits with the change or not at all. Stopping the server right after a person is added can no longer lose the process that the addition starts, or start it twice.
2. Tasks and processes become shared foundations in core/, beside records, and import no business module. Features plug into them.

Processes start on domain events in the same transaction

Until now, processes started only on a record's date, through the daily job. Event triggers were refused because events were dispatched in memory after the commit, and a restart lost them.

• Trigger on domain events. A process definition can use { "kind": "on_event", "event": "employees.added", "when": { "field": "to", "in": ["done", "cancelled"] }, "run": "in_same_transaction", "faultMode": "rollback_transaction" }. when filters on one payload field. run and faultMode are required, so that running after the commit and failing quietly can be added later without breaking definitions.
• Before-commit handlers. The transactor hands each round of a transaction's new events, each with an id, to before-commit handlers. A round's handlers can publish more events, up to 10 rounds. Past that the change fails, and the error names the event types that kept coming. A handler's error rolls the change back. After-commit handlers run as before.
• One run per event and record, batched. Every matching event starts a run for each record it is about. A person added in a 5,000-person import gets their own run, but the runs, tasks and steps of a save are inserted in a fixed number of statements, in batches of 1,000. A record whose run of the process is still in progress, or that is gone, is skipped. A run that cannot start fails the change, and the log names the process and the events.
• Task events. Tasks publish tasks.task_created and tasks.task_status_changed, with byEmployeeId, the person who acted. A run keeps the event as it concerns its own record (process_runs.trigger_event), so a step can tell who made the change.
• No new waits. Event starts take no process lock. The existing process locks become FOR NO KEY UPDATE, which still makes the daily job, an admin's start, step commands and resets take turns without blocking a new run. A reset locks every task before it deletes employees, so it waits for an in-flight task change instead of deadlocking with it.
• The daily job and an admin's start now use the same batched start. No shipped process uses an event trigger yet.

Tasks and processes become core

• Record types register what a process needs from their records: who a record is about (readSubjects) and the dates a record-date trigger may start on (dateFields, findByDateBetween). Employees registers employee (its startDate); tasks registers task.
• Triggering events are registered by the module that publishes them (employees.added by employees; tasks.task_created and tasks.task_status_changed by tasks) and collected once at startup. A key registered twice stops startup.
• People and working days are read through PeoplePort and WorkingCalendarsPort, which employees fills with adapters, together with the bank holiday rule. Ports live in a module's ports/ folder and their implementations in the implementing module's adapters/ folder; lint enforces both as layers.
• A process step's task is marked done with the ordinary tasks.markDone. Processes listens to the task's status event before the commit and completes the step and the run in the same transaction. Every transaction that locks a task and its run takes the task first; the daily check now locks a run's open tasks before the run. A run that can no longer go on (its record or person is gone) is left for the daily check, and a task done meanwhile stays done.
• Tasks no longer store which process step created them. Processes serves Task.processSource and employees serves the employee fields of Task, so the GraphQL shape the web app reads stays the same.
• Tasks and runs are domain aggregates. Task owns who sees it, who marks it done or deletes it, and that done and cancelled are final. ProcessRun holds its steps and owns completing them, finishing, the removal check and cancelling. Changes load it with lockRun and write it back with saveRun.
• Process definitions. A step keeps key and kind at its root and its kind's fields under it (task: { assignee, title, description, due }). startableByAdmin is gone, so an admin may start any process by hand with create on process_run. The run steps table is now process_run_step_progress: the steps live in the version's definition, and the table holds a run's progress on each.
• Counting working days moves to utils (addWorkingDays); processes keeps what a day offset means and its limit.
• New rule RULE77: a module.md opens with JSON examples of its main flow. processes/module.md shows a run started by a change and by a date.

Behaviour changes

• Done and cancelled tasks are final: Reopen, the Undo of Mark done and the reopen step command are gone.
• People no longer cancel tasks. The person who asked for an open task can delete it (tasks.delete), after a confirmation. Only a process cancels the tasks it created, with a reason.
• Cancelling a run by hand is gone from the process service (it had no API or screen).

Testing

• Integration tests for event starts:
  • Adding a person commits exactly one run and its task.
  • A rolled-back change leaves no run.
  • One event for three people, and three events in one save, each start three runs. 5,000 people in one event start 5,000 runs.
  • A run that cannot start (no bank holidays known for the year) rolls back adding the person.
  • A process that is off, or a when that does not match, starts nothing.
  • Marking a task done starts a run that carries the person who acted; marking it waiting does not.
  • A run's tasks start another process in the same save.
  • A matching event during a run in progress is skipped, and the save commits.
  • An event start completes while another transaction holds the process lock.
  • A reset waits for a task change whose event starts a run, and removes that run.
  • Core tests cover the rounds, the 10-round limit and rollback.
• Integration tests for tasks and processes as core:
  • Deleting a task: by the person who asked, the assignee and an unrelated caller, and a done, a cancelled and a process task.
  • Finishing steps through tasks.markDone: both steps, one step, concurrent completion, a removed subject or assignee, and manual tasks left untouched.
  • The daily check cancels a run whose task record was deleted.
  • Definitions are parsed against registrations, and a key registered twice is refused.
• pnpm check, all API integration tests and all end-to-end tests pass; web unit tests pass.
• The migrations are edited in place; drizzle-kit generate reports no change for tasks and processes. The Docker image builds, and pnpm dev starts on a fresh database.
• docs/qa/tasks.md is updated for Delete and for the removed Reopen and Undo.
• Manually, with test processes added in the database, on pnpm dev and on the Docker Compose image:
  • Adding a person, killing the API with SIGKILL right after the response and restarting it leaves exactly one run and one task.
  • A broken process fails the add, with nothing saved.
  • Marking a task done starts its process; reopening the task does not.
  • A self-triggering process stops after 10 rounds and rolls back.
  • Filling the demo with a process on starts 40 runs in one batch in 0.24 s, and Exit demo clears them.
  • Five concurrent adds all commit with their runs.
• Not done: screenshots of the Delete menu and dialog.

Release Notes:

• Improved: marking a process task done now works from Tasks like any other task.
• Changed: a task you asked for can be deleted while it is open; done tasks can no longer be reopened, and tasks can no longer be cancelled by hand.